TensorFlow Estimators: Train, Evaluate, Export, Explained!

The purporse of this tutorial is to explain how TensorFlow estimator trainining and evaluation work with different configurations, as well as how the model is exported for serving. The tutorial covers the following points:

Creating an estimator using the premade DNNClassifier for the Census dataset.
Parameterizing input function for training.
Training with incremental steps vs. total Steps.
Training with steps vs. epochs.
Controling checkpoints creation frequency.
Parameterizing input function for evaluation
Interwining training and evaluation.
Creating a serving input function for exporting the model
Configuring Latest exporter, final exporter, and best exporter



In [2]:

    
import os
import pandas as pd
import numpy as np
from datetime import datetime

import tensorflow as tf
from tensorflow import data

print("TensorFlow : {}".format(tf.__version__))

SEED = 19831060









    



TensorFlow : 1.12.0

Download Data



In [7]:

    
DATA_DIR='data'
!mkdir $DATA_DIR
!gsutil cp gs://cloud-samples-data/ml-engine/census/data/adult.data.csv $DATA_DIR
!gsutil cp gs://cloud-samples-data/ml-engine/census/data/adult.test.csv $DATA_DIR









    



mkdir: data: File exists
Copying gs://cloud-samples-data/ml-engine/census/data/adult.data.csv...
- [1 files][  3.8 MiB/  3.8 MiB]                                                
Operation completed over 1 objects/3.8 MiB.                                      
Copying gs://cloud-samples-data/ml-engine/census/data/adult.test.csv...
- [1 files][  1.9 MiB/  1.9 MiB]                                                
Operation completed over 1 objects/1.9 MiB.



In [8]:

    
TRAIN_DATA_FILE = os.path.join(DATA_DIR, 'adult.data.csv')
EVAL_DATA_FILE = os.path.join(DATA_DIR, 'adult.test.csv')



In [19]:

    
!wc -l $TRAIN_DATA_FILE
!wc -l $EVAL_DATA_FILE









    



   32561 data/adult.data.csv
   16278 data/adult.test.csv

The training data includes 32561 records, while the evaluation data includes 16278 records. We will fix the batch size to 200



In [20]:

    
TRAIN_DATA_SIZE = 32561
EVAL_DATA_SIZE = 16278
BATCH_SIZE = 200

print("Training data size:{}".format(TRAIN_DATA_SIZE))
print("Number of batches in training data: {}".format(TRAIN_DATA_SIZE/float(BATCH_SIZE)))
print("Evaluation data size:{}".format(EVAL_DATA_SIZE))
print("Number of batches in evaluation data: {}".format(EVAL_DATA_SIZE/float(BATCH_SIZE)))









    



Training data size:32561
Number of batches in training data: 162.805
Evaluation data size:16278
Number of batches in evaluation data: 81.39

Dataset Metadata



In [21]:

    
HEADER = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',
               'marital_status', 'occupation', 'relationship', 'race', 'gender',
               'capital_gain', 'capital_loss', 'hours_per_week',
               'native_country', 'income_bracket']

HEADER_DEFAULTS = [[0], [''], [0], [''], [0], [''], [''], [''], [''], [''],
                       [0], [0], [0], [''], ['']]

NUMERIC_FEATURE_NAMES = ['age', 'education_num', 'capital_gain', 'capital_loss', 'hours_per_week']
CATEGORICAL_FEATURE_NAMES = ['gender', 'race', 'education', 'marital_status', 'relationship', 
                             'workclass', 'occupation', 'native_country']

FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES
TARGET_NAME = 'income_bracket'
TARGET_LABELS = [' <=50K', ' >50K']
WEIGHT_COLUMN_NAME = 'fnlwgt'

def get_categorical_features_vocabolary():
    data = pd.read_csv(TRAIN_DATA_FILE, names=HEADER)
    return {
        column: list(data[column].unique()) 
        for column in data.columns if column in CATEGORICAL_FEATURE_NAMES
    }



In [22]:

    
feature_vocabolary = get_categorical_features_vocabolary()
print(feature_vocabolary)









    



{'workclass': [' State-gov', ' Self-emp-not-inc', ' Private', ' Federal-gov', ' Local-gov', ' ?', ' Self-emp-inc', ' Without-pay', ' Never-worked'], 'relationship': [' Not-in-family', ' Husband', ' Wife', ' Own-child', ' Unmarried', ' Other-relative'], 'gender': [' Male', ' Female'], 'marital_status': [' Never-married', ' Married-civ-spouse', ' Divorced', ' Married-spouse-absent', ' Separated', ' Married-AF-spouse', ' Widowed'], 'race': [' White', ' Black', ' Asian-Pac-Islander', ' Amer-Indian-Eskimo', ' Other'], 'native_country': [' United-States', ' Cuba', ' Jamaica', ' India', ' ?', ' Mexico', ' South', ' Puerto-Rico', ' Honduras', ' England', ' Canada', ' Germany', ' Iran', ' Philippines', ' Italy', ' Poland', ' Columbia', ' Cambodia', ' Thailand', ' Ecuador', ' Laos', ' Taiwan', ' Haiti', ' Portugal', ' Dominican-Republic', ' El-Salvador', ' France', ' Guatemala', ' China', ' Japan', ' Yugoslavia', ' Peru', ' Outlying-US(Guam-USVI-etc)', ' Scotland', ' Trinadad&Tobago', ' Greece', ' Nicaragua', ' Vietnam', ' Hong', ' Ireland', ' Hungary', ' Holand-Netherlands'], 'education': [' Bachelors', ' HS-grad', ' 11th', ' Masters', ' 9th', ' Some-college', ' Assoc-acdm', ' Assoc-voc', ' 7th-8th', ' Doctorate', ' Prof-school', ' 5th-6th', ' 10th', ' 1st-4th', ' Preschool', ' 12th'], 'occupation': [' Adm-clerical', ' Exec-managerial', ' Handlers-cleaners', ' Prof-specialty', ' Other-service', ' Sales', ' Craft-repair', ' Transport-moving', ' Farming-fishing', ' Machine-op-inspct', ' Tech-support', ' ?', ' Protective-serv', ' Armed-Forces', ' Priv-house-serv']}

Create Estimator

Note that, the purpose of the tutorial is not to create the state-of-the-art model for this dataset. The purpose is to show the mechanisms of training and evaluating a TensorFlow estimator, regardless of the sophistication of the model or its predictive power. Thus, we use a premade tf.estimator.DNNClassifier for our examples. The ideas discussed in this tutorial applies to more complex custom estimators.



In [23]:

    
import math

def create_feature_columns():
    
    feature_columns = []
    
    for column in NUMERIC_FEATURE_NAMES:
        feature_column = tf.feature_column.numeric_column(column)
        feature_columns.append(feature_column)
        
    for column in CATEGORICAL_FEATURE_NAMES:
        vocabolary = feature_vocabolary[column]
        embed_size = int(math.sqrt(len(vocabolary)))
        feature_column = tf.feature_column.embedding_column(
            tf.feature_column.categorical_column_with_vocabulary_list(column, vocabolary), 
            embed_size)
        feature_columns.append(feature_column)
        
    return feature_columns
                  

def create_estimator(run_config):
    
    feature_columns = create_feature_columns()
    
    estimator = tf.estimator.DNNClassifier(
        feature_columns=feature_columns,
        n_classes=len(TARGET_LABELS),
        label_vocabulary=TARGET_LABELS,
        weight_column=WEIGHT_COLUMN_NAME,
        hidden_units=[100, 70, 50] ,
        dropout=0.2,
        batch_norm=True,
        config=run_config
    )
    
    return estimator

MODELS_LOCATION = 'models/census'
MODEL_NAME = 'dnn_classifier'
model_dir = os.path.join(MODELS_LOCATION, MODEL_NAME)

print(model_dir)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    model_dir=model_dir
)









    



models/census/dnn_classifier

Data Input Function



In [3]:

    
def make_input_fn(file_pattern, batch_size, num_epochs, shuffle=False):
    
    def _input_fn():
        dataset = tf.data.experimental.make_csv_dataset(
            file_pattern=file_pattern,
            batch_size=batch_size,
            column_names=HEADER,
            column_defaults=HEADER_DEFAULTS,
            label_name=TARGET_NAME,
            field_delim=',',
            use_quote_delim=True,
            header=False,
            num_epochs=num_epochs,
            shuffle=shuffle
        )
        return dataset
    
    return _input_fn

Train: Input Function

Batch size is set
Epochs is ignored (set to None)

Later we are going to see how to use epochs for training.



In [25]:

    
train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=None,
    shuffle=True
)

Train: Incremental Steps vs. Total Steps

1 batch (feed forward pass & backpropagation) corresponds to 1 training step
steps: Number of steps for which to train model. 'steps' works incrementally. Two calls to train(steps=100) means 200 training iterations.
max_steps: Number of total steps for which to train model. If set, steps must be None. Two calls to train(max_steps=100) means that the second call will not do any iteration since first call did all 100 steps.

In the following function, clean_start flag indicates whether to delete the previous model artefacts (if any), and incremental flag indicates whether to use steps (for incremental training steps) or max_steps (for overall training steps).



In [26]:

    
def train_experiment(training_steps, clean_start, incremental, run_config):

    if clean_start == True: 
        if tf.gfile.Exists(run_config.model_dir):
            print("Removing previous artefacts...")
            
            tf.gfile.DeleteRecursively(run_config.model_dir)

    print("")
    estimator = create_estimator(run_config)
    print("")
    
    time_start = datetime.utcnow() 
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................") 
   
    if incremental:
        # Use steps parameter
        estimator.train(train_input_fn, steps=training_steps)
    else:
        # Use max_steps parameter
        estimator.train(train_input_fn, max_steps=training_steps)
        
    time_end = datetime.utcnow() 
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))
    
    return estimator



In [27]:

    
train_experiment(
    training_steps=1000, 
    clean_start=True,
    incremental=False,
    run_config=run_config
)









    



Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x10ca58ed0>, '_model_dir': 'models/census/dnn_classifier', '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': 19831060, '_save_summary_steps': 100, '_device_fn': None, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_global_id_in_cluster': 0, '_master': ''}

Experiment started at 16:52:35
.......................................
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 31132416.0, step = 1
INFO:tensorflow:global_step/sec: 82.2509
INFO:tensorflow:loss = 12670475.0, step = 101 (1.218 sec)
INFO:tensorflow:global_step/sec: 169.578
INFO:tensorflow:loss = 11341477.0, step = 201 (0.590 sec)
INFO:tensorflow:global_step/sec: 177.614
INFO:tensorflow:loss = 12852321.0, step = 301 (0.563 sec)
INFO:tensorflow:global_step/sec: 163.928
INFO:tensorflow:loss = 13684520.0, step = 401 (0.610 sec)
INFO:tensorflow:global_step/sec: 169.234
INFO:tensorflow:loss = 12090486.0, step = 501 (0.591 sec)
INFO:tensorflow:global_step/sec: 187.021
INFO:tensorflow:loss = 13600504.0, step = 601 (0.534 sec)
INFO:tensorflow:global_step/sec: 167.494
INFO:tensorflow:loss = 14767286.0, step = 701 (0.597 sec)
INFO:tensorflow:global_step/sec: 145.886
INFO:tensorflow:loss = 10702760.0, step = 801 (0.685 sec)
INFO:tensorflow:global_step/sec: 153.083
INFO:tensorflow:loss = 13668747.0, step = 901 (0.654 sec)
INFO:tensorflow:Saving checkpoints for 1000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Loss for final step: 14524872.0.
.......................................
Experiment finished at 16:52:47

Experiment elapsed time: 12.109969 seconds






    Out[27]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x105902610>

Total number of steps 1000.

Lets run this again, with max_steps, without deleting the previous model.



In [87]:

    
train_experiment(
    training_steps=1000, 
    clean_start=False,
    incremental=False,
    run_config=run_config,
)









    



INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19831006, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11fb97150>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 16:33:57
.......................................
INFO:tensorflow:Skipping training since max_steps has already saved.
.......................................
Experiment finished at 16:33:57

Experiment elapsed time: 0.008284 seconds






    Out[87]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x121f3b890>

As expected, no training occured and since max_steps was reached.

Now let's try incremetal steps



In [88]:

    
train_experiment(
    training_steps=1000, 
    clean_start=False,
    incremental=True,
    run_config=run_config
)









    



INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19831006, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11fb97150>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 16:34:05
.......................................
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 1000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 70.22403, step = 1001
INFO:tensorflow:global_step/sec: 53.6648
INFO:tensorflow:loss = 65.78857, step = 1101 (1.866 sec)
INFO:tensorflow:global_step/sec: 135.459
INFO:tensorflow:loss = 63.059696, step = 1201 (0.738 sec)
INFO:tensorflow:global_step/sec: 133.357
INFO:tensorflow:loss = 61.91352, step = 1301 (0.750 sec)
INFO:tensorflow:global_step/sec: 129.434
INFO:tensorflow:loss = 70.51839, step = 1401 (0.772 sec)
INFO:tensorflow:global_step/sec: 97.7296
INFO:tensorflow:loss = 70.49428, step = 1501 (1.023 sec)
INFO:tensorflow:global_step/sec: 97.3648
INFO:tensorflow:loss = 74.98505, step = 1601 (1.027 sec)
INFO:tensorflow:global_step/sec: 115.098
INFO:tensorflow:loss = 53.296, step = 1701 (0.869 sec)
INFO:tensorflow:global_step/sec: 122.998
INFO:tensorflow:loss = 65.77577, step = 1801 (0.813 sec)
INFO:tensorflow:global_step/sec: 118.097
INFO:tensorflow:loss = 60.06965, step = 1901 (0.847 sec)
INFO:tensorflow:Saving checkpoints for 2000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Loss for final step: 65.54395.
.......................................
Experiment finished at 16:34:21

Experiment elapsed time: 15.772218 seconds






    Out[88]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x1088a2650>

As shown, the total number of training steps is 2000, starting from step 1000 (from the previous run)

Train: Steps vs Epochs

While the steps refers to how many data batchs are needed for training, the epochs refers to how many times the whole training data needs to be used for training.

While using epochs to define the number of training iteration is a conventional practice in machine learninig, however, when working with very large datasets to train Deep Learning models, batch-level training steps (rather than the whole-training-data-level epochs) are more practical.



In [106]:

    
num_epochs=3

train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=num_epochs,
    shuffle=True,
)



In [109]:

    
expected_training_steps = math.ceil(TRAIN_DATA_SIZE/float(BATCH_SIZE))*num_epochs

print('Training data size: {}'.format(TRAIN_DATA_SIZE))
print('Batch size: {}'.format(BATCH_SIZE) )
print('Number of epochs (supplied): {}'.format(num_epochs))
print('Number of training steps (expected): {}'.format(expected_training_steps))
print('')

train_experiment(
    training_steps=None, 
    clean_start=True,
    incremental=True,
    run_config=run_config
)









    



Training data size: 32561
Batch size: 200
Number of epochs (supplied): 3
Number of training steps (expected): 489.0

Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19831006, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11fb97150>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 16:45:32
.......................................
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 165.60706, step = 1
INFO:tensorflow:global_step/sec: 55.2379
INFO:tensorflow:loss = 74.62894, step = 101 (1.813 sec)
INFO:tensorflow:global_step/sec: 127.153
INFO:tensorflow:loss = 70.498184, step = 201 (0.786 sec)
INFO:tensorflow:global_step/sec: 130.382
INFO:tensorflow:loss = 66.00288, step = 301 (0.768 sec)
INFO:tensorflow:global_step/sec: 135.813
INFO:tensorflow:loss = 72.86255, step = 401 (0.736 sec)
INFO:tensorflow:Saving checkpoints for 489 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Loss for final step: 30.459267.
.......................................
Experiment finished at 16:45:44

Experiment elapsed time: 11.901956 seconds






    Out[109]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11f699f10>

As expected, the training steps, given 3 epochs (for training data of 32561 records and batch size of 200 records), is 32561, which correspods to: ceiling of (TRAIN_DATA_SIZE / BATCH_SIZE) * num_epochs)

Note that, if both num_epochs (in the train_input_fn) and steps (in estimator.train) are supplied, the model will stop on the earlier criteria.



In [110]:

    
train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=1000,
    shuffle=True,
)

train_experiment(
    training_steps=10, # the model will train for only 10 steps, ignoring the 1000 epochs
    clean_start=True,
    incremental=True,
    run_config=run_config
)









    



Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19831006, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11fb97150>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 16:47:35
.......................................
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 165.60706, step = 1
INFO:tensorflow:Saving checkpoints for 10 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Loss for final step: 96.85506.
.......................................
Experiment finished at 16:47:42

Experiment elapsed time: 6.871136 seconds






    Out[110]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11453d250>



In [111]:

    
train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=1, # the model will train for only 1 epoch (55 steps), ignoring the 1000 steps
    shuffle=True,
)

train_experiment(
    training_steps=1000, 
    clean_start=True,
    incremental=True,
    run_config=run_config
)









    



Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19831006, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11fb97150>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 16:48:20
.......................................
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 173.16132, step = 1
INFO:tensorflow:global_step/sec: 54.4522
INFO:tensorflow:loss = 82.83362, step = 101 (1.839 sec)
INFO:tensorflow:Saving checkpoints for 163 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Loss for final step: 55.227993.
.......................................
Experiment finished at 16:48:28

Experiment elapsed time: 8.290039 seconds






    Out[111]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11e135110>

Train: Checkpoints

By default, a checkpoint is saved every 600 secs (10mins). This behaviour is configured in the run_config passed to the estimator, using only one of the following parameters:

save_checkpoints_secs: Save checkpoints every this many seconds.
save_checkpoints_steps: Save checkpoints every this many steps.

In addition, you can specify the number of the checkpoints to keep using keep_checkpoint_max Defaults to 5 (that is, the 5 most recent checkpoint files are kept.)

The following code trains the model for 1000 steps...



In [114]:

    
os.environ['MODEL_DIR'] = model_dir



In [112]:

    
train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=None,
    shuffle=True,
)

train_experiment(
    training_steps=1000, 
    clean_start=True,
    incremental=True,
    run_config=run_config # using the default checkpoints param values
)









    



Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19831006, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11fb97150>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 16:49:03
.......................................
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 165.60706, step = 1
INFO:tensorflow:global_step/sec: 57.9846
INFO:tensorflow:loss = 74.62894, step = 101 (1.727 sec)
INFO:tensorflow:global_step/sec: 140.082
INFO:tensorflow:loss = 70.498184, step = 201 (0.714 sec)
INFO:tensorflow:global_step/sec: 133.418
INFO:tensorflow:loss = 66.00288, step = 301 (0.752 sec)
INFO:tensorflow:global_step/sec: 128.202
INFO:tensorflow:loss = 72.86255, step = 401 (0.778 sec)
INFO:tensorflow:global_step/sec: 130.34
INFO:tensorflow:loss = 72.79736, step = 501 (0.768 sec)
INFO:tensorflow:global_step/sec: 129.013
INFO:tensorflow:loss = 75.737885, step = 601 (0.774 sec)
INFO:tensorflow:global_step/sec: 118.185
INFO:tensorflow:loss = 51.82579, step = 701 (0.846 sec)
INFO:tensorflow:global_step/sec: 131.568
INFO:tensorflow:loss = 66.95524, step = 801 (0.760 sec)
INFO:tensorflow:global_step/sec: 123.295
INFO:tensorflow:loss = 61.824047, step = 901 (0.812 sec)
INFO:tensorflow:Saving checkpoints for 1000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Loss for final step: 68.06041.
.......................................
Experiment finished at 16:49:18

Experiment elapsed time: 15.32673 seconds






    Out[112]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11ef36390>



In [115]:

    
%%bash

ls ${MODEL_DIR}









    



checkpoint
graph.pbtxt
model.ckpt-0.data-00000-of-00001
model.ckpt-0.index
model.ckpt-0.meta
model.ckpt-1000.data-00000-of-00001
model.ckpt-1000.index
model.ckpt-1000.meta

As shown, since the training (1000 iterasion) finished in less than 600 seconds (default value for **save_checkpoint_sec), only 2 checkpoints where saved: the initial one, and the final one.

Now let's set save_checkpoints_steps in the run_config to 200, so that in 1000 steps, you produce 5 checkpoints



In [116]:

    
run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    model_dir=model_dir,
    save_checkpoints_steps=200, ## so in 1000 steps, you produce 5 checkpoints
    save_checkpoints_secs=None
)



In [117]:

    
estimator=train_experiment(
    training_steps=1000, 
    clean_start=True,
    incremental=True,
    run_config=run_config 
)









    



Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 198301006, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11f8d4f90>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': 200, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 16:51:27
.......................................
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 191.36717, step = 1
INFO:tensorflow:global_step/sec: 57.8952
INFO:tensorflow:loss = 79.96772, step = 101 (1.729 sec)
INFO:tensorflow:Saving checkpoints for 200 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:global_step/sec: 95.7056
INFO:tensorflow:loss = 77.50678, step = 201 (1.045 sec)
INFO:tensorflow:global_step/sec: 128.854
INFO:tensorflow:loss = 68.35478, step = 301 (0.777 sec)
INFO:tensorflow:Saving checkpoints for 400 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:global_step/sec: 93.2821
INFO:tensorflow:loss = 83.95572, step = 401 (1.072 sec)
INFO:tensorflow:global_step/sec: 83.9175
INFO:tensorflow:loss = 65.67826, step = 501 (1.192 sec)
INFO:tensorflow:Saving checkpoints for 600 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:global_step/sec: 63.7453
INFO:tensorflow:loss = 74.13786, step = 601 (1.569 sec)
INFO:tensorflow:global_step/sec: 113.99
INFO:tensorflow:loss = 65.84137, step = 701 (0.876 sec)
INFO:tensorflow:Saving checkpoints for 800 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:global_step/sec: 86.3981
INFO:tensorflow:loss = 69.787704, step = 801 (1.158 sec)
INFO:tensorflow:global_step/sec: 134.801
INFO:tensorflow:loss = 68.7874, step = 901 (0.741 sec)
INFO:tensorflow:Saving checkpoints for 1000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Loss for final step: 69.935394.
.......................................
Experiment finished at 16:51:45

Experiment elapsed time: 17.872937 seconds



In [118]:

    
%%bash

ls ${MODEL_DIR}









    



checkpoint
graph.pbtxt
model.ckpt-1000.data-00000-of-00001
model.ckpt-1000.index
model.ckpt-1000.meta
model.ckpt-200.data-00000-of-00001
model.ckpt-200.index
model.ckpt-200.meta
model.ckpt-400.data-00000-of-00001
model.ckpt-400.index
model.ckpt-400.meta
model.ckpt-600.data-00000-of-00001
model.ckpt-600.index
model.ckpt-600.meta
model.ckpt-800.data-00000-of-00001
model.ckpt-800.index
model.ckpt-800.meta

Each checkpoint is labelled by the step number it was saved in.

Evaluate: Epochs vs Steps

batch_size is set (which can be bigger than batch size of training, of the batch fits in memory)
num_epochs is usually set to 1 (as you want to evaluate your model on the entire evaluation data once)



In [122]:

    
eval_input_fn = make_input_fn(
    EVAL_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=1,
    shuffle=False,
)

For evaluation, if you set epochs to be 1, you can ignore the steps param (set it you None).

By default, the latest checkpoint is evaluated



In [123]:

    
estimator.evaluate(eval_input_fn, steps=None)









    



INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-16:52:48
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-16:52:50
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.7857231, accuracy_baseline = 0.7637916, auc = 0.8588228, auc_precision_recall = 0.64802057, average_loss = 0.47923627, global_step = 1000, label/mean = 0.23620838, loss = 95.13425, precision = 0.8635438, prediction/mean = 0.13552067, recall = 0.110273086
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: models/census/dnn_classifier/model.ckpt-1000






    Out[123]:





{'accuracy': 0.7857231,
 'accuracy_baseline': 0.7637916,
 'auc': 0.8588228,
 'auc_precision_recall': 0.64802057,
 'average_loss': 0.47923627,
 'global_step': 1000,
 'label/mean': 0.23620838,
 'loss': 95.13425,
 'precision': 0.8635438,
 'prediction/mean': 0.13552067,
 'recall': 0.110273086}

This is equivalent to setting epochs to None and setting the steps to the number of batches in the dataset

Export: Serving Input Receiver Function



In [125]:

    
def make_serving_input_receiver_fn():
    
    inputs = {}
    for feature_name in FEATURE_NAMES:
        dtype = tf.float32 if feature_name in NUMERIC_FEATURE_NAMES else tf.string
        inputs[feature_name] = tf.placeholder(shape=[None], dtype=dtype)
        
    return tf.estimator.export.build_raw_serving_input_receiver_fn(inputs)

export_dir = os.path.join(model_dir, 'export')

if tf.gfile.Exists(export_dir):
    tf.gfile.DeleteRecursively(export_dir)
        
estimator.export_savedmodel(
    export_dir_base=export_dir,
    serving_input_receiver_fn=make_serving_input_receiver_fn()
)









    



INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_10:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_9:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_5:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_8:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_6:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_12:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_7:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_11:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_10:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_9:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_5:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_8:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_6:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_12:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_7:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_11:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_10:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_9:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_5:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_8:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_6:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_12:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_7:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_11:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/temp-1547571440/saved_model.pb






    Out[125]:





'models/census/dnn_classifier/export/1547571440'



In [126]:

    
%%bash

saved_models_base=${MODEL_DIR}/export/
saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo ${saved_model_dir}
ls ${saved_model_dir}
saved_model_cli show --dir=${saved_model_dir} --all









    



models/census/dnn_classifier/export/1547571440
saved_model.pb
variables

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['predict']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['age'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder:0
    inputs['capital_gain'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_2:0
    inputs['capital_loss'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_3:0
    inputs['education'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_7:0
    inputs['education_num'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_1:0
    inputs['gender'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_5:0
    inputs['hours_per_week'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_4:0
    inputs['marital_status'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_8:0
    inputs['native_country'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_12:0
    inputs['occupation'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_11:0
    inputs['race'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_6:0
    inputs['relationship'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_9:0
    inputs['workclass'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_10:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['class_ids'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 1)
        name: dnn/head/predictions/ExpandDims:0
    outputs['classes'] tensor_info:
        dtype: DT_STRING
        shape: (-1, 1)
        name: dnn/head/predictions/class_string_lookup_Lookup:0
    outputs['logistic'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: dnn/head/predictions/logistic:0
    outputs['logits'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: dnn/logits/BiasAdd:0
    outputs['probabilities'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 2)
        name: dnn/head/predictions/probabilities:0
  Method name is: tensorflow/serving/predict

Intertwining Training & Evalution

Use TrainSpec & EvalSpec with tf.estimator.train_and_evaluate()
In TrainSpec:
- num_epochs in the train_input_fn is ignored. (Set it to None)
- You need to set max_steps param, otherwise it will train forever
In EvalSpec (to evaluate the model using the whole evaluation data once):
- num_epochs is set to 1 in the eval_input_fn
- steps param is set to None
Evaluation occurs when a new checkpoint is saved
Checkpoints saving frequency is configures in run_config (using save_checkpoints_steps or save_checkpoints_secs)
You can set minimum amount of time between two evaluation, using throttle_secs in EvalSpec. For example, if throttle_secs is set to 60sec, this means that the following evaluation will only occure after 60sec from the previous evaluation, even if save_checkpoins_sec is set to 10.
If throttle_secs is set to 0, then evaluation will occure each time a checkpoint is saved, regardless the time difference between two consequtive checkpoints



In [134]:

    
def train_and_evaluate_experiment(params, run_config):
    
    # TrainSpec ####################################
    train_input_fn = make_input_fn(
        TRAIN_DATA_FILE,
        batch_size=BATCH_SIZE,
        num_epochs=None,
        shuffle=True
    )
    
    train_spec = tf.estimator.TrainSpec(
        input_fn = train_input_fn,
        max_steps=params.traning_steps
    )
    ###############################################
    
    
    # EvalSpec ####################################
    eval_input_fn = make_input_fn(
        EVAL_DATA_FILE,
        batch_size=BATCH_SIZE,
        num_epochs=1,
        shuffle=False
    )

    eval_spec = tf.estimator.EvalSpec(
        name=datetime.utcnow().strftime("%H%M%S"),
        input_fn = eval_input_fn,
        steps=None,
        start_delay_secs=0,
        throttle_secs=params.eval_throttle_secs
    )
    
    ###############################################

    tf.logging.set_verbosity(tf.logging.INFO)
    
    if params.clean_start:
        if tf.gfile.Exists(run_config.model_dir):
            print("Removing previous artefacts...")
            tf.gfile.DeleteRecursively(run_config.model_dir)
            

    print("")
    estimator = create_estimator(run_config)
    print("")
    
    time_start = datetime.utcnow() 
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................") 

    tf.estimator.train_and_evaluate(
        estimator=estimator,
        train_spec=train_spec, 
        eval_spec=eval_spec
    )

    time_end = datetime.utcnow() 
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))
    
    return estimator

Now let's try the following:

Training for 1000 steps (set num_epochs to None and max_steps to 1000).
Save a checkpoint after each 200 steps (set save_checkpoints_steps to 200).
Evaluate when each checkpoint is produced (set eval_throttle_secs to 0). That is, 5 evaluations in total
Keep only the latest 3 checkpoints out of the 5 checkpoints to be saved (set keep_checkpoint_max to 3)
When evaluating, use the whole eval_data once (set num_epochs to 1 and steps to None).



In [135]:

    
params  = tf.contrib.training.HParams(
    batch_size=BATCH_SIZE,
    traning_steps=1000,
    eval_throttle_secs=0,
    clean_start=True,
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=200,
    keep_checkpoint_max=3,
    model_dir=model_dir
)



In [136]:

    
train_and_evaluate_experiment(params, run_config)









    



Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 3, '_tf_random_seed': 19830610, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11e3c9b90>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': 200, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 17:37:36
.......................................
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 200 or save_checkpoints_secs None.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 155.29642, step = 1
INFO:tensorflow:global_step/sec: 56.5761
INFO:tensorflow:loss = 67.47958, step = 101 (1.770 sec)
INFO:tensorflow:Saving checkpoints for 200 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:37:49
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-200
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:37:51
INFO:tensorflow:Saving dict for global step 200: accuracy = 0.7627473, accuracy_baseline = 0.7637916, auc = 0.8826315, auc_precision_recall = 0.70138216, average_loss = 0.5482385, global_step = 200, label/mean = 0.23620838, loss = 108.83203, precision = 0.49876148, prediction/mean = 0.47243804, recall = 0.89024705
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 200: models/census/dnn_classifier/model.ckpt-200
INFO:tensorflow:global_step/sec: 13.0679
INFO:tensorflow:loss = 81.95731, step = 201 (7.654 sec)
INFO:tensorflow:global_step/sec: 74.9237
INFO:tensorflow:loss = 72.70906, step = 301 (1.333 sec)
INFO:tensorflow:Saving checkpoints for 400 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:37:57
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-400
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:37:59
INFO:tensorflow:Saving dict for global step 400: accuracy = 0.80341566, accuracy_baseline = 0.7637916, auc = 0.8757313, auc_precision_recall = 0.67542946, average_loss = 0.39762452, global_step = 400, label/mean = 0.23620838, loss = 78.93331, precision = 0.6526266, prediction/mean = 0.24921176, recall = 0.35864758
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 400: models/census/dnn_classifier/model.ckpt-400
INFO:tensorflow:global_step/sec: 16.2411
INFO:tensorflow:loss = 64.828674, step = 401 (6.158 sec)
INFO:tensorflow:global_step/sec: 112.436
INFO:tensorflow:loss = 69.630424, step = 501 (0.888 sec)
INFO:tensorflow:Saving checkpoints for 600 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:38:04
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-600
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:38:07
INFO:tensorflow:Saving dict for global step 600: accuracy = 0.80403, accuracy_baseline = 0.7637916, auc = 0.883371, auc_precision_recall = 0.6964862, average_loss = 0.37735343, global_step = 600, label/mean = 0.23620838, loss = 74.909256, precision = 0.79162955, prediction/mean = 0.2035477, recall = 0.23120937
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 600: models/census/dnn_classifier/model.ckpt-600
INFO:tensorflow:global_step/sec: 13.1135
INFO:tensorflow:loss = 59.467163, step = 601 (7.627 sec)
INFO:tensorflow:global_step/sec: 130.678
INFO:tensorflow:loss = 53.453083, step = 701 (0.765 sec)
INFO:tensorflow:Saving checkpoints for 800 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:38:12
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-800
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:38:13
INFO:tensorflow:Saving dict for global step 800: accuracy = 0.82030964, accuracy_baseline = 0.7637916, auc = 0.88977325, auc_precision_recall = 0.7128348, average_loss = 0.37313807, global_step = 800, label/mean = 0.23620838, loss = 74.07246, precision = 0.8412463, prediction/mean = 0.20912503, recall = 0.2949285
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 800: models/census/dnn_classifier/model.ckpt-800
INFO:tensorflow:global_step/sec: 19.4176
INFO:tensorflow:loss = 73.53612, step = 801 (5.149 sec)
INFO:tensorflow:global_step/sec: 123.452
INFO:tensorflow:loss = 65.87018, step = 901 (0.810 sec)
INFO:tensorflow:Saving checkpoints for 1000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:38:18
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:38:20
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.8172994, accuracy_baseline = 0.7637916, auc = 0.88677675, auc_precision_recall = 0.7010293, average_loss = 0.38544405, global_step = 1000, label/mean = 0.23620838, loss = 76.51534, precision = 0.86021507, prediction/mean = 0.16257516, recall = 0.27048114
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: models/census/dnn_classifier/model.ckpt-1000
INFO:tensorflow:Loss for final step: 71.52243.
.......................................
Experiment finished at 17:38:20

Experiment elapsed time: 43.771883 seconds






    Out[136]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x12180ff90>



In [137]:

    
%%bash

ls ${MODEL_DIR}









    



checkpoint
eval_173736
graph.pbtxt
model.ckpt-1000.data-00000-of-00001
model.ckpt-1000.index
model.ckpt-1000.meta
model.ckpt-600.data-00000-of-00001
model.ckpt-600.index
model.ckpt-600.meta
model.ckpt-800.data-00000-of-00001
model.ckpt-800.index
model.ckpt-800.meta

In order to train the model for num_epochs, you need to do the following:

the training data size needs to be known before training
compute the training steps as: (TRAIN_DATA_SIZE / BATCH_SIZE) * num_epochs
In TrainSpec, max_step to the computed value
set num_epochs in the train_input_fn to None



In [142]:

    
num_epochs = 5
computed_traning_steps = int(math.ceil((TRAIN_DATA_SIZE/float(BATCH_SIZE)))*num_epochs)

print('Training data size: {}'.format(TRAIN_DATA_SIZE))
print('Batch size: {}'.format(BATCH_SIZE))
print('Number of epochs (supplied): {}'.format(BATCH_SIZE)) 
print('Number of training steps (computed): {}'.format(computed_traning_steps))
print('')


params = tf.contrib.training.HParams(
    batch_size=BATCH_SIZE,
    traning_steps=computed_traning_steps,
    eval_throttle_secs=0,
    clean_start=True,
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=200,
    model_dir=model_dir
)

train_and_evaluate_experiment(params, run_config)









    



Training data size: 32561
Batch size: 200
Number of epochs (supplied): 200
Number of training steps (computed): 815

Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19830610, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11a8771d0>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': 200, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 17:42:28
.......................................
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 200 or save_checkpoints_secs None.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 155.29642, step = 1
INFO:tensorflow:global_step/sec: 51.5617
INFO:tensorflow:loss = 67.47958, step = 101 (1.941 sec)
INFO:tensorflow:Saving checkpoints for 200 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:42:41
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-200
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:42:43
INFO:tensorflow:Saving dict for global step 200: accuracy = 0.7627473, accuracy_baseline = 0.7637916, auc = 0.8826315, auc_precision_recall = 0.70138216, average_loss = 0.5482385, global_step = 200, label/mean = 0.23620838, loss = 108.83203, precision = 0.49876148, prediction/mean = 0.47243804, recall = 0.89024705
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 200: models/census/dnn_classifier/model.ckpt-200
INFO:tensorflow:global_step/sec: 16.4177
INFO:tensorflow:loss = 81.95731, step = 201 (6.092 sec)
INFO:tensorflow:global_step/sec: 101.897
INFO:tensorflow:loss = 72.70906, step = 301 (0.981 sec)
INFO:tensorflow:Saving checkpoints for 400 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:42:48
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-400
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:42:50
INFO:tensorflow:Saving dict for global step 400: accuracy = 0.80341566, accuracy_baseline = 0.7637916, auc = 0.8757313, auc_precision_recall = 0.67542946, average_loss = 0.39762452, global_step = 400, label/mean = 0.23620838, loss = 78.93331, precision = 0.6526266, prediction/mean = 0.24921176, recall = 0.35864758
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 400: models/census/dnn_classifier/model.ckpt-400
INFO:tensorflow:global_step/sec: 16.9094
INFO:tensorflow:loss = 64.828674, step = 401 (5.914 sec)
INFO:tensorflow:global_step/sec: 116.078
INFO:tensorflow:loss = 69.630424, step = 501 (0.861 sec)
INFO:tensorflow:Saving checkpoints for 600 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:42:54
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-600
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:42:56
INFO:tensorflow:Saving dict for global step 600: accuracy = 0.80403, accuracy_baseline = 0.7637916, auc = 0.883371, auc_precision_recall = 0.6964862, average_loss = 0.37735343, global_step = 600, label/mean = 0.23620838, loss = 74.909256, precision = 0.79162955, prediction/mean = 0.2035477, recall = 0.23120937
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 600: models/census/dnn_classifier/model.ckpt-600
INFO:tensorflow:global_step/sec: 18.2909
INFO:tensorflow:loss = 59.467163, step = 601 (5.469 sec)
INFO:tensorflow:global_step/sec: 110.414
INFO:tensorflow:loss = 53.453083, step = 701 (0.904 sec)
INFO:tensorflow:Saving checkpoints for 800 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:43:00
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-800
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:43:03
INFO:tensorflow:Saving dict for global step 800: accuracy = 0.82030964, accuracy_baseline = 0.7637916, auc = 0.88977325, auc_precision_recall = 0.7128348, average_loss = 0.37313807, global_step = 800, label/mean = 0.23620838, loss = 74.07246, precision = 0.8412463, prediction/mean = 0.20912503, recall = 0.2949285
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 800: models/census/dnn_classifier/model.ckpt-800
INFO:tensorflow:global_step/sec: 18.9892
INFO:tensorflow:loss = 73.53612, step = 801 (5.266 sec)
INFO:tensorflow:Saving checkpoints for 815 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:43:05
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-815
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:43:07
INFO:tensorflow:Saving dict for global step 815: accuracy = 0.79911536, accuracy_baseline = 0.7637916, auc = 0.88614124, auc_precision_recall = 0.6890154, average_loss = 0.38953334, global_step = 815, label/mean = 0.23620838, loss = 77.32712, precision = 0.8016789, prediction/mean = 0.17416908, recall = 0.19869961
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 815: models/census/dnn_classifier/model.ckpt-815
INFO:tensorflow:Loss for final step: 69.640785.
.......................................
Experiment finished at 17:43:07

Experiment elapsed time: 39.204086 seconds






    Out[142]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11a3b6b90>

Note that, since we set save_checkpoint_steps to 200, and we have 815 steps, we get 5 checkpoints (and corresponding evaluations):

at step 200
at step 400
at step 600
at step 800
in the end (at step 815)



In [143]:

    
%%bash

ls ${MODEL_DIR}









    



checkpoint
eval_174228
graph.pbtxt
model.ckpt-200.data-00000-of-00001
model.ckpt-200.index
model.ckpt-200.meta
model.ckpt-400.data-00000-of-00001
model.ckpt-400.index
model.ckpt-400.meta
model.ckpt-600.data-00000-of-00001
model.ckpt-600.index
model.ckpt-600.meta
model.ckpt-800.data-00000-of-00001
model.ckpt-800.index
model.ckpt-800.meta
model.ckpt-815.data-00000-of-00001
model.ckpt-815.index
model.ckpt-815.meta

Note that the checkpoint at the beginning of the training is removed, since keep_checkpoint_max is set to 5 (default value).

Train, Evaluate, and Export



In [147]:

    
def train_evaluate_export_experiment(params, run_config, exporter):
    
    # TrainSpec ####################################
    train_input_fn = make_input_fn(
        TRAIN_DATA_FILE,
        batch_size=BATCH_SIZE,
        num_epochs=None,
        shuffle=True
    )
    
    train_spec = tf.estimator.TrainSpec(
        input_fn = train_input_fn,
        max_steps=params.traning_steps
    )
    ###############################################
    
    
    # EvalSpec ####################################
    eval_input_fn = make_input_fn(
        EVAL_DATA_FILE,
        batch_size=BATCH_SIZE,
        num_epochs=1,
        shuffle=False
    )
    
    eval_spec = tf.estimator.EvalSpec(
        name=params.eval_name,
        input_fn=eval_input_fn,
        exporters=[exporter],
        steps=None,
        start_delay_secs=0,
        throttle_secs=params.eval_throttle_secs
    )
    ###############################################

    tf.logging.set_verbosity(tf.logging.INFO)
    
    if params.clean_start:
        if tf.gfile.Exists(run_config.model_dir):
            print("Removing previous artefacts...")
            tf.gfile.DeleteRecursively(run_config.model_dir)
            

    print("")
    estimator = create_estimator(run_config)
    print("")
    
    time_start = datetime.utcnow() 
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................") 

    tf.estimator.train_and_evaluate(
        estimator=estimator,
        train_spec=train_spec, 
        eval_spec=eval_spec
    )

    time_end = datetime.utcnow() 
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))
    
    return estimator

Latest exporter

This exports a model after each evaluation.

You can specify the maximum number of exported models to keep using exports_to_keep param



In [148]:

    
exporter = tf.estimator.LatestExporter(
            name="estimate", 
            serving_input_receiver_fn=make_serving_input_receiver_fn(),
            exports_to_keep=3,
)

params = tf.contrib.training.HParams(
    batch_size=BATCH_SIZE,
    traning_steps=computed_traning_steps,
    eval_throttle_secs=0,
    clean_start=True,
    eval_name=datetime.utcnow().strftime("%H%M%S")
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=200,
    model_dir=model_dir
)

train_evaluate_export_experiment(params, run_config, exporter)









    



Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19830610, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11e2a05d0>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': 250, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 17:48:58
.......................................
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 250 or save_checkpoints_secs None.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 155.29642, step = 1
INFO:tensorflow:global_step/sec: 47.4137
INFO:tensorflow:loss = 67.47958, step = 101 (2.111 sec)
INFO:tensorflow:global_step/sec: 132.992
INFO:tensorflow:loss = 81.95731, step = 201 (0.752 sec)
INFO:tensorflow:Saving checkpoints for 250 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:49:10
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-250
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:49:12
INFO:tensorflow:Saving dict for global step 250: accuracy = 0.82245976, accuracy_baseline = 0.7637916, auc = 0.86175174, auc_precision_recall = 0.6733779, average_loss = 0.46543393, global_step = 250, label/mean = 0.23620838, loss = 92.39431, precision = 0.62048954, prediction/mean = 0.34760657, recall = 0.63953185
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 250: models/census/dnn_classifier/model.ckpt-250
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-250
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547574553/saved_model.pb
INFO:tensorflow:global_step/sec: 13.7444
INFO:tensorflow:loss = 72.70906, step = 301 (7.277 sec)
INFO:tensorflow:global_step/sec: 123.796
INFO:tensorflow:loss = 64.828674, step = 401 (0.808 sec)
INFO:tensorflow:Saving checkpoints for 500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:49:19
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:49:21
INFO:tensorflow:Saving dict for global step 500: accuracy = 0.8076545, accuracy_baseline = 0.7637916, auc = 0.88104653, auc_precision_recall = 0.6930925, average_loss = 0.37480116, global_step = 500, label/mean = 0.23620838, loss = 74.4026, precision = 0.703303, prediction/mean = 0.22721802, recall = 0.32119635
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 500: models/census/dnn_classifier/model.ckpt-500
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-500
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547574561/saved_model.pb
INFO:tensorflow:global_step/sec: 14.2357
INFO:tensorflow:loss = 69.630424, step = 501 (7.026 sec)
INFO:tensorflow:global_step/sec: 122.027
INFO:tensorflow:loss = 59.467163, step = 601 (0.819 sec)
INFO:tensorflow:global_step/sec: 117.493
INFO:tensorflow:loss = 53.453083, step = 701 (0.850 sec)
INFO:tensorflow:Saving checkpoints for 750 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:49:28
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-750
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:49:30
INFO:tensorflow:Saving dict for global step 750: accuracy = 0.8150264, accuracy_baseline = 0.7637916, auc = 0.8850932, auc_precision_recall = 0.7017188, average_loss = 0.37720585, global_step = 750, label/mean = 0.23620838, loss = 74.87996, precision = 0.8116592, prediction/mean = 0.20266667, recall = 0.28244475
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 750: models/census/dnn_classifier/model.ckpt-750
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-750
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547574570/saved_model.pb
INFO:tensorflow:global_step/sec: 12.9171
INFO:tensorflow:loss = 73.53612, step = 801 (7.743 sec)
INFO:tensorflow:Saving checkpoints for 815 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:49:35
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-815
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:49:37
INFO:tensorflow:Saving dict for global step 815: accuracy = 0.79911536, accuracy_baseline = 0.7637916, auc = 0.88614124, auc_precision_recall = 0.6890154, average_loss = 0.38953334, global_step = 815, label/mean = 0.23620838, loss = 77.32712, precision = 0.8016789, prediction/mean = 0.17416908, recall = 0.19869961
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 815: models/census/dnn_classifier/model.ckpt-815
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_56:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_62:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_61:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_57:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_52:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_60:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_58:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_54:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_64:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_55:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_59:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_53:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_63:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-815
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547574577/saved_model.pb
INFO:tensorflow:Loss for final step: 69.640785.
.......................................
Experiment finished at 17:49:38

Experiment elapsed time: 40.548516 seconds






    Out[148]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x121920650>



In [149]:

    
%%bash

saved_models_base=${MODEL_DIR}/export/estimate/
echo 'exported model folders:'
ls ${saved_models_base}
echo ''

saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo 'last exported model: '${saved_model_dir}
ls ${saved_model_dir}
saved_model_cli show --dir=${saved_model_dir} --all









    



exported model folders:
1547574561
1547574570
1547574577

last exported model: models/census/dnn_classifier/export/estimate/1547574577
saved_model.pb
variables

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['predict']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['age'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_52:0
    inputs['capital_gain'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_54:0
    inputs['capital_loss'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_55:0
    inputs['education'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_59:0
    inputs['education_num'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_53:0
    inputs['gender'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_57:0
    inputs['hours_per_week'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_56:0
    inputs['marital_status'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_60:0
    inputs['native_country'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_64:0
    inputs['occupation'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_63:0
    inputs['race'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_58:0
    inputs['relationship'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_61:0
    inputs['workclass'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_62:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['class_ids'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 1)
        name: dnn/head/predictions/ExpandDims:0
    outputs['classes'] tensor_info:
        dtype: DT_STRING
        shape: (-1, 1)
        name: dnn/head/predictions/class_string_lookup_Lookup:0
    outputs['logistic'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: dnn/head/predictions/logistic:0
    outputs['logits'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: dnn/logits/BiasAdd:0
    outputs['probabilities'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 2)
        name: dnn/head/predictions/probabilities:0
  Method name is: tensorflow/serving/predict

Final exporter

This exports only the very last evaluated checkpoint of the model



In [150]:

    
exporter = tf.estimator.FinalExporter(
            name="estimate",
            serving_input_receiver_fn=make_serving_input_receiver_fn()
)

params = tf.contrib.training.HParams(
    batch_size=BATCH_SIZE,
    traning_steps=computed_traning_steps,
    eval_throttle_secs=0,
    clean_start=True,
    eval_name=datetime.utcnow().strftime("%H%M%S")
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=200,
    model_dir=model_dir
)

train_evaluate_export_experiment(params, run_config, exporter)









    



Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19830610, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11a266d90>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': 250, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 17:50:16
.......................................
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 250 or save_checkpoints_secs None.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 155.29642, step = 1
INFO:tensorflow:global_step/sec: 47.4339
INFO:tensorflow:loss = 67.47958, step = 101 (2.110 sec)
INFO:tensorflow:global_step/sec: 97.4946
INFO:tensorflow:loss = 81.95731, step = 201 (1.025 sec)
INFO:tensorflow:Saving checkpoints for 250 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:50:30
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-250
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:50:33
INFO:tensorflow:Saving dict for global step 250: accuracy = 0.82245976, accuracy_baseline = 0.7637916, auc = 0.86175174, auc_precision_recall = 0.6733779, average_loss = 0.46543393, global_step = 250, label/mean = 0.23620838, loss = 92.39431, precision = 0.62048954, prediction/mean = 0.34760657, recall = 0.63953185
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 250: models/census/dnn_classifier/model.ckpt-250
INFO:tensorflow:global_step/sec: 13.9511
INFO:tensorflow:loss = 72.70906, step = 301 (7.170 sec)
INFO:tensorflow:global_step/sec: 82.049
INFO:tensorflow:loss = 64.828674, step = 401 (1.217 sec)
INFO:tensorflow:Saving checkpoints for 500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:50:39
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:50:41
INFO:tensorflow:Saving dict for global step 500: accuracy = 0.8076545, accuracy_baseline = 0.7637916, auc = 0.88104653, auc_precision_recall = 0.6930925, average_loss = 0.37480116, global_step = 500, label/mean = 0.23620838, loss = 74.4026, precision = 0.703303, prediction/mean = 0.22721802, recall = 0.32119635
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 500: models/census/dnn_classifier/model.ckpt-500
INFO:tensorflow:global_step/sec: 15.1202
INFO:tensorflow:loss = 69.630424, step = 501 (6.617 sec)
INFO:tensorflow:global_step/sec: 96.3977
INFO:tensorflow:loss = 59.467163, step = 601 (1.034 sec)
INFO:tensorflow:global_step/sec: 81.6115
INFO:tensorflow:loss = 53.453083, step = 701 (1.225 sec)
INFO:tensorflow:Saving checkpoints for 750 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:50:47
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-750
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:50:49
INFO:tensorflow:Saving dict for global step 750: accuracy = 0.8150264, accuracy_baseline = 0.7637916, auc = 0.8850932, auc_precision_recall = 0.7017188, average_loss = 0.37720585, global_step = 750, label/mean = 0.23620838, loss = 74.87996, precision = 0.8116592, prediction/mean = 0.20266667, recall = 0.28244475
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 750: models/census/dnn_classifier/model.ckpt-750
INFO:tensorflow:global_step/sec: 17.3622
INFO:tensorflow:loss = 73.53612, step = 801 (5.760 sec)
INFO:tensorflow:Saving checkpoints for 815 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:50:52
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-815
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:50:55
INFO:tensorflow:Saving dict for global step 815: accuracy = 0.79911536, accuracy_baseline = 0.7637916, auc = 0.88614124, auc_precision_recall = 0.6890154, average_loss = 0.38953334, global_step = 815, label/mean = 0.23620838, loss = 77.32712, precision = 0.8016789, prediction/mean = 0.17416908, recall = 0.19869961
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 815: models/census/dnn_classifier/model.ckpt-815
INFO:tensorflow:Performing the final export in the end of training.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_69:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_75:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_74:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_70:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_65:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_73:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_71:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_67:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_77:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_68:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_72:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_66:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_76:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_69:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_75:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_74:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_70:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_65:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_73:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_71:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_67:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_77:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_68:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_72:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_66:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_76:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_69:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_75:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_74:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_70:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_65:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_73:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_71:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_67:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_77:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_68:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_72:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_66:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_76:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-815
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547574655/saved_model.pb
INFO:tensorflow:Loss for final step: 69.640785.
.......................................
Experiment finished at 17:50:57

Experiment elapsed time: 40.933479 seconds






    Out[150]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11a266390>



In [151]:

    
%%bash

saved_models_base=${MODEL_DIR}/export/estimate/
echo 'exported model folders:'
ls ${saved_models_base}
echo ''

saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo 'last exported model: '${saved_model_dir}
ls ${saved_model_dir}
saved_model_cli show --dir=${saved_model_dir} --all









    



exported model folders:
1547574655

last exported model: models/census/dnn_classifier/export/estimate/1547574655
saved_model.pb
variables

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['predict']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['age'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_65:0
    inputs['capital_gain'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_67:0
    inputs['capital_loss'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_68:0
    inputs['education'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_72:0
    inputs['education_num'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_66:0
    inputs['gender'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_70:0
    inputs['hours_per_week'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_69:0
    inputs['marital_status'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_73:0
    inputs['native_country'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_77:0
    inputs['occupation'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_76:0
    inputs['race'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_71:0
    inputs['relationship'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_74:0
    inputs['workclass'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_75:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['class_ids'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 1)
        name: dnn/head/predictions/ExpandDims:0
    outputs['classes'] tensor_info:
        dtype: DT_STRING
        shape: (-1, 1)
        name: dnn/head/predictions/class_string_lookup_Lookup:0
    outputs['logistic'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: dnn/head/predictions/logistic:0
    outputs['logits'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: dnn/logits/BiasAdd:0
    outputs['probabilities'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 2)
        name: dnn/head/predictions/probabilities:0
  Method name is: tensorflow/serving/predict

Best exporter

This runs everytime when the new model is better than any exsiting model.

It uses the evaluation events stored under the eval folder.

You need to set the name of the subfolder in the EvalSpec, and set the event_file_pattern in the BestExporter to point to this folder and perform the evalution comparesions.



In [157]:

    
eval_name=datetime.utcnow().strftime("%H%M%S")

exporter = tf.estimator.BestExporter(
    event_file_pattern='eval_{}/*.tfevents.*'.format(eval_name),
    name="estimate", 
    serving_input_receiver_fn=make_serving_input_receiver_fn(),
    exports_to_keep=1
)

params = tf.contrib.training.HParams(
    batch_size=BATCH_SIZE,
    traning_steps=10000,
    eval_throttle_secs=0,
    exporter_type='best',
    clean_start=True,
    eval_name=eval_name
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=500,
    model_dir=model_dir
)

train_evaluate_export_experiment(params, run_config, exporter)









    



Removing previous artefacts...

INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_global_id_in_cluster': 0, '_session_config': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 19831060, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x124afb850>, '_model_dir': 'models/census/dnn_classifier', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_master': '', '_save_checkpoints_steps': 500, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_device_fn': None, '_save_summary_steps': 100, '_num_ps_replicas': 0}

Experiment started at 17:56:27
.......................................
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 500 or save_checkpoints_secs None.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:loss = 172.95993, step = 1
INFO:tensorflow:global_step/sec: 52.4609
INFO:tensorflow:loss = 71.77466, step = 101 (1.909 sec)
INFO:tensorflow:global_step/sec: 115.047
INFO:tensorflow:loss = 58.99911, step = 201 (0.868 sec)
INFO:tensorflow:global_step/sec: 114.344
INFO:tensorflow:loss = 60.97943, step = 301 (0.874 sec)
INFO:tensorflow:global_step/sec: 116.479
INFO:tensorflow:loss = 78.19871, step = 401 (0.859 sec)
INFO:tensorflow:Saving checkpoints for 500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:56:42
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:56:44
INFO:tensorflow:Saving dict for global step 500: accuracy = 0.8107261, accuracy_baseline = 0.7637916, auc = 0.8853111, auc_precision_recall = 0.69354385, average_loss = 0.36734027, global_step = 500, label/mean = 0.23620838, loss = 72.921524, precision = 0.73785806, prediction/mean = 0.2016623, recall = 0.30819246
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 500: models/census/dnn_classifier/model.ckpt-500
INFO:tensorflow:Loading best metric from event files.
INFO:tensorflow:global_step/sec: 15.3281
INFO:tensorflow:loss = 68.5647, step = 501 (6.525 sec)
INFO:tensorflow:global_step/sec: 122.489
INFO:tensorflow:loss = 62.382122, step = 601 (0.816 sec)
INFO:tensorflow:global_step/sec: 100.351
INFO:tensorflow:loss = 90.863434, step = 701 (0.996 sec)
INFO:tensorflow:global_step/sec: 111.907
INFO:tensorflow:loss = 54.618965, step = 801 (0.894 sec)
INFO:tensorflow:global_step/sec: 110.464
INFO:tensorflow:loss = 75.02841, step = 901 (0.908 sec)
INFO:tensorflow:Saving checkpoints for 1000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:56:52
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:56:54
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.7859074, accuracy_baseline = 0.7637916, auc = 0.8885201, auc_precision_recall = 0.6920164, average_loss = 0.39756644, global_step = 1000, label/mean = 0.23620838, loss = 78.92178, precision = 0.9390244, prediction/mean = 0.1465797, recall = 0.10013004
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: models/census/dnn_classifier/model.ckpt-1000
INFO:tensorflow:global_step/sec: 18.1089
INFO:tensorflow:loss = 77.07351, step = 1001 (5.519 sec)
INFO:tensorflow:global_step/sec: 118.122
INFO:tensorflow:loss = 67.64369, step = 1101 (0.848 sec)
INFO:tensorflow:global_step/sec: 114.609
INFO:tensorflow:loss = 65.69208, step = 1201 (0.872 sec)
INFO:tensorflow:global_step/sec: 112.475
INFO:tensorflow:loss = 73.88888, step = 1301 (0.888 sec)
INFO:tensorflow:global_step/sec: 113.318
INFO:tensorflow:loss = 69.1914, step = 1401 (0.883 sec)
INFO:tensorflow:Saving checkpoints for 1500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:57:01
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-1500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:57:03
INFO:tensorflow:Saving dict for global step 1500: accuracy = 0.8036614, accuracy_baseline = 0.7637916, auc = 0.89025176, auc_precision_recall = 0.69514704, average_loss = 0.36788926, global_step = 1500, label/mean = 0.23620838, loss = 73.0305, precision = 0.83282053, prediction/mean = 0.18293026, recall = 0.21118335
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1500: models/census/dnn_classifier/model.ckpt-1500
INFO:tensorflow:global_step/sec: 17.938
INFO:tensorflow:loss = 72.47092, step = 1501 (5.576 sec)
INFO:tensorflow:global_step/sec: 120.105
INFO:tensorflow:loss = 67.17369, step = 1601 (0.833 sec)
INFO:tensorflow:global_step/sec: 114.814
INFO:tensorflow:loss = 67.22575, step = 1701 (0.871 sec)
INFO:tensorflow:global_step/sec: 116.934
INFO:tensorflow:loss = 83.45018, step = 1801 (0.855 sec)
INFO:tensorflow:global_step/sec: 112.811
INFO:tensorflow:loss = 68.75552, step = 1901 (0.887 sec)
INFO:tensorflow:Saving checkpoints for 2000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:57:10
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-2000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:57:12
INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.79696524, accuracy_baseline = 0.7637916, auc = 0.89136416, auc_precision_recall = 0.70869017, average_loss = 0.3759436, global_step = 2000, label/mean = 0.23620838, loss = 74.62939, precision = 0.8739612, prediction/mean = 0.17577307, recall = 0.16410923
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2000: models/census/dnn_classifier/model.ckpt-2000
INFO:tensorflow:global_step/sec: 18.3033
INFO:tensorflow:loss = 53.46354, step = 2001 (5.462 sec)
INFO:tensorflow:global_step/sec: 117.603
INFO:tensorflow:loss = 69.60867, step = 2101 (0.851 sec)
INFO:tensorflow:global_step/sec: 111.021
INFO:tensorflow:loss = 66.10107, step = 2201 (0.901 sec)
INFO:tensorflow:global_step/sec: 114.271
INFO:tensorflow:loss = 77.2023, step = 2301 (0.876 sec)
INFO:tensorflow:global_step/sec: 111.396
INFO:tensorflow:loss = 65.51532, step = 2401 (0.897 sec)
INFO:tensorflow:Saving checkpoints for 2500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:57:19
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-2500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:57:22
INFO:tensorflow:Saving dict for global step 2500: accuracy = 0.7984396, accuracy_baseline = 0.7637916, auc = 0.8953998, auc_precision_recall = 0.7283441, average_loss = 0.37381795, global_step = 2500, label/mean = 0.23620838, loss = 74.20743, precision = 0.9028571, prediction/mean = 0.17095402, recall = 0.16436931
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2500: models/census/dnn_classifier/model.ckpt-2500
INFO:tensorflow:global_step/sec: 16.8272
INFO:tensorflow:loss = 69.35632, step = 2501 (5.947 sec)
INFO:tensorflow:global_step/sec: 84.2024
INFO:tensorflow:loss = 61.520332, step = 2601 (1.183 sec)
INFO:tensorflow:global_step/sec: 89.8955
INFO:tensorflow:loss = 65.93899, step = 2701 (1.113 sec)
INFO:tensorflow:global_step/sec: 76.1069
INFO:tensorflow:loss = 68.2942, step = 2801 (1.315 sec)
INFO:tensorflow:global_step/sec: 77.9262
INFO:tensorflow:loss = 67.17973, step = 2901 (1.284 sec)
INFO:tensorflow:Saving checkpoints for 3000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:57:31
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-3000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:57:33
INFO:tensorflow:Saving dict for global step 3000: accuracy = 0.8021256, accuracy_baseline = 0.7637916, auc = 0.896139, auc_precision_recall = 0.72200584, average_loss = 0.3665196, global_step = 3000, label/mean = 0.23620838, loss = 72.75861, precision = 0.88423645, prediction/mean = 0.1953986, recall = 0.18673602
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3000: models/census/dnn_classifier/model.ckpt-3000
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-3000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575053/saved_model.pb
INFO:tensorflow:global_step/sec: 12.4251
INFO:tensorflow:loss = 63.866314, step = 3001 (8.046 sec)
INFO:tensorflow:global_step/sec: 118.801
INFO:tensorflow:loss = 64.46978, step = 3101 (0.841 sec)
INFO:tensorflow:global_step/sec: 104.76
INFO:tensorflow:loss = 59.73033, step = 3201 (0.954 sec)
INFO:tensorflow:global_step/sec: 83.7591
INFO:tensorflow:loss = 64.31961, step = 3301 (1.195 sec)
INFO:tensorflow:global_step/sec: 90.3256
INFO:tensorflow:loss = 65.04874, step = 3401 (1.107 sec)
INFO:tensorflow:Saving checkpoints for 3500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:57:43
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-3500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:57:45
INFO:tensorflow:Saving dict for global step 3500: accuracy = 0.80267847, accuracy_baseline = 0.7637916, auc = 0.8970522, auc_precision_recall = 0.7322527, average_loss = 0.37224555, global_step = 3500, label/mean = 0.23620838, loss = 73.89528, precision = 0.9031847, prediction/mean = 0.16327314, recall = 0.18439531
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3500: models/census/dnn_classifier/model.ckpt-3500
INFO:tensorflow:global_step/sec: 16.7232
INFO:tensorflow:loss = 61.537792, step = 3501 (5.980 sec)
INFO:tensorflow:global_step/sec: 90.3517
INFO:tensorflow:loss = 64.823944, step = 3601 (1.106 sec)
INFO:tensorflow:global_step/sec: 79.041
INFO:tensorflow:loss = 53.05521, step = 3701 (1.266 sec)
INFO:tensorflow:global_step/sec: 100.916
INFO:tensorflow:loss = 63.682304, step = 3801 (0.991 sec)
INFO:tensorflow:global_step/sec: 78.4977
INFO:tensorflow:loss = 62.415936, step = 3901 (1.274 sec)
INFO:tensorflow:Saving checkpoints for 4000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:57:53
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-4000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:57:56
INFO:tensorflow:Saving dict for global step 4000: accuracy = 0.80771595, accuracy_baseline = 0.7637916, auc = 0.8941131, auc_precision_recall = 0.71038806, average_loss = 0.364013, global_step = 4000, label/mean = 0.23620838, loss = 72.26102, precision = 0.84675074, prediction/mean = 0.18409665, recall = 0.22704811
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4000: models/census/dnn_classifier/model.ckpt-4000
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-4000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575076/saved_model.pb
INFO:tensorflow:global_step/sec: 12.6673
INFO:tensorflow:loss = 73.917984, step = 4001 (7.895 sec)
INFO:tensorflow:global_step/sec: 104.434
INFO:tensorflow:loss = 72.05383, step = 4101 (0.956 sec)
INFO:tensorflow:global_step/sec: 93.5788
INFO:tensorflow:loss = 54.487705, step = 4201 (1.068 sec)
INFO:tensorflow:global_step/sec: 105.509
INFO:tensorflow:loss = 64.41385, step = 4301 (0.949 sec)
INFO:tensorflow:global_step/sec: 95.2062
INFO:tensorflow:loss = 80.535194, step = 4401 (1.049 sec)
INFO:tensorflow:Saving checkpoints for 4500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:58:05
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-4500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:58:08
INFO:tensorflow:Saving dict for global step 4500: accuracy = 0.8065487, accuracy_baseline = 0.7637916, auc = 0.89536446, auc_precision_recall = 0.7228568, average_loss = 0.36613637, global_step = 4500, label/mean = 0.23620838, loss = 72.68253, precision = 0.8580247, prediction/mean = 0.17809206, recall = 0.21690507
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4500: models/census/dnn_classifier/model.ckpt-4500
INFO:tensorflow:global_step/sec: 15.3798
INFO:tensorflow:loss = 67.30796, step = 4501 (6.503 sec)
INFO:tensorflow:global_step/sec: 106.209
INFO:tensorflow:loss = 69.88489, step = 4601 (0.941 sec)
INFO:tensorflow:global_step/sec: 101.647
INFO:tensorflow:loss = 56.048347, step = 4701 (0.984 sec)
INFO:tensorflow:global_step/sec: 112.683
INFO:tensorflow:loss = 69.424194, step = 4801 (0.887 sec)
INFO:tensorflow:global_step/sec: 103.779
INFO:tensorflow:loss = 54.095524, step = 4901 (0.963 sec)
INFO:tensorflow:Saving checkpoints for 5000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:58:15
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-5000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:58:18
INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.81963384, accuracy_baseline = 0.7637916, auc = 0.8945979, auc_precision_recall = 0.7114821, average_loss = 0.35755154, global_step = 5000, label/mean = 0.23620838, loss = 70.97834, precision = 0.7554806, prediction/mean = 0.2260002, recall = 0.34954485
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: models/census/dnn_classifier/model.ckpt-5000
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-5000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575098/saved_model.pb
INFO:tensorflow:global_step/sec: 12.0818
INFO:tensorflow:loss = 55.372242, step = 5001 (8.278 sec)
INFO:tensorflow:global_step/sec: 120.01
INFO:tensorflow:loss = 62.12554, step = 5101 (0.832 sec)
INFO:tensorflow:global_step/sec: 115.688
INFO:tensorflow:loss = 74.47404, step = 5201 (0.864 sec)
INFO:tensorflow:global_step/sec: 86.0244
INFO:tensorflow:loss = 71.59109, step = 5301 (1.164 sec)
INFO:tensorflow:global_step/sec: 87.2262
INFO:tensorflow:loss = 62.195507, step = 5401 (1.146 sec)
INFO:tensorflow:Saving checkpoints for 5500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:58:28
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-5500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:58:30
INFO:tensorflow:Saving dict for global step 5500: accuracy = 0.82786584, accuracy_baseline = 0.7637916, auc = 0.8963913, auc_precision_recall = 0.72098964, average_loss = 0.3481061, global_step = 5500, label/mean = 0.23620838, loss = 69.1033, precision = 0.7550122, prediction/mean = 0.21262617, recall = 0.40156046
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5500: models/census/dnn_classifier/model.ckpt-5500
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-5500
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575110/saved_model.pb
INFO:tensorflow:global_step/sec: 12.2815
INFO:tensorflow:loss = 65.16937, step = 5501 (8.142 sec)
INFO:tensorflow:global_step/sec: 118.259
INFO:tensorflow:loss = 68.17366, step = 5601 (0.846 sec)
INFO:tensorflow:global_step/sec: 93.4147
INFO:tensorflow:loss = 58.278008, step = 5701 (1.071 sec)
INFO:tensorflow:global_step/sec: 94.4329
INFO:tensorflow:loss = 65.88693, step = 5801 (1.058 sec)
INFO:tensorflow:global_step/sec: 88.5387
INFO:tensorflow:loss = 71.134476, step = 5901 (1.130 sec)
INFO:tensorflow:Saving checkpoints for 6000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:58:40
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-6000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:58:42
INFO:tensorflow:Saving dict for global step 6000: accuracy = 0.81490356, accuracy_baseline = 0.7637916, auc = 0.89778954, auc_precision_recall = 0.7317071, average_loss = 0.35727024, global_step = 6000, label/mean = 0.23620838, loss = 70.9225, precision = 0.8312102, prediction/mean = 0.18713109, recall = 0.27152145
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 6000: models/census/dnn_classifier/model.ckpt-6000
INFO:tensorflow:global_step/sec: 15.4228
INFO:tensorflow:loss = 79.257095, step = 6001 (6.483 sec)
INFO:tensorflow:global_step/sec: 95.3104
INFO:tensorflow:loss = 59.182884, step = 6101 (1.050 sec)
INFO:tensorflow:global_step/sec: 95.9991
INFO:tensorflow:loss = 91.768524, step = 6201 (1.063 sec)
INFO:tensorflow:global_step/sec: 103.843
INFO:tensorflow:loss = 61.51592, step = 6301 (0.941 sec)
INFO:tensorflow:global_step/sec: 94.8085
INFO:tensorflow:loss = 60.018642, step = 6401 (1.055 sec)
INFO:tensorflow:Saving checkpoints for 6500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:58:50
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-6500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:58:52
INFO:tensorflow:Saving dict for global step 6500: accuracy = 0.8383708, accuracy_baseline = 0.7637916, auc = 0.8997856, auc_precision_recall = 0.7365786, average_loss = 0.3458071, global_step = 6500, label/mean = 0.23620838, loss = 68.64693, precision = 0.7804991, prediction/mean = 0.22430843, recall = 0.43927178
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 6500: models/census/dnn_classifier/model.ckpt-6500
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-6500
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575132/saved_model.pb
INFO:tensorflow:global_step/sec: 13.1399
INFO:tensorflow:loss = 63.19798, step = 6501 (7.611 sec)
INFO:tensorflow:global_step/sec: 113.346
INFO:tensorflow:loss = 53.464714, step = 6601 (0.881 sec)
INFO:tensorflow:global_step/sec: 112.769
INFO:tensorflow:loss = 74.693504, step = 6701 (0.888 sec)
INFO:tensorflow:global_step/sec: 110.501
INFO:tensorflow:loss = 54.94541, step = 6801 (0.905 sec)
INFO:tensorflow:global_step/sec: 92.3131
INFO:tensorflow:loss = 62.869537, step = 6901 (1.083 sec)
INFO:tensorflow:Saving checkpoints for 7000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:59:02
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-7000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:59:04
INFO:tensorflow:Saving dict for global step 7000: accuracy = 0.83861655, accuracy_baseline = 0.7637916, auc = 0.8997396, auc_precision_recall = 0.7368864, average_loss = 0.345184, global_step = 7000, label/mean = 0.23620838, loss = 68.52323, precision = 0.8107143, prediction/mean = 0.21117531, recall = 0.41326398
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 7000: models/census/dnn_classifier/model.ckpt-7000
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-7000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575144/saved_model.pb
INFO:tensorflow:global_step/sec: 14.7704
INFO:tensorflow:loss = 66.00528, step = 7001 (6.770 sec)
INFO:tensorflow:global_step/sec: 137.83
INFO:tensorflow:loss = 63.14566, step = 7101 (0.727 sec)
INFO:tensorflow:global_step/sec: 125.061
INFO:tensorflow:loss = 73.543495, step = 7201 (0.798 sec)
INFO:tensorflow:global_step/sec: 124.074
INFO:tensorflow:loss = 61.756073, step = 7301 (0.806 sec)
INFO:tensorflow:global_step/sec: 119.592
INFO:tensorflow:loss = 48.584934, step = 7401 (0.837 sec)
INFO:tensorflow:Saving checkpoints for 7500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:59:11
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-7500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:59:13
INFO:tensorflow:Saving dict for global step 7500: accuracy = 0.8493058, accuracy_baseline = 0.7637916, auc = 0.8981279, auc_precision_recall = 0.7397051, average_loss = 0.3412308, global_step = 7500, label/mean = 0.23620838, loss = 67.73848, precision = 0.7457627, prediction/mean = 0.22837831, recall = 0.54928476
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 7500: models/census/dnn_classifier/model.ckpt-7500
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-7500
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575153/saved_model.pb
INFO:tensorflow:global_step/sec: 14.0799
INFO:tensorflow:loss = 64.476776, step = 7501 (7.103 sec)
INFO:tensorflow:global_step/sec: 103.861
INFO:tensorflow:loss = 56.446087, step = 7601 (0.962 sec)
INFO:tensorflow:global_step/sec: 120.781
INFO:tensorflow:loss = 73.36093, step = 7701 (0.829 sec)
INFO:tensorflow:global_step/sec: 114.779
INFO:tensorflow:loss = 66.01928, step = 7801 (0.870 sec)
INFO:tensorflow:global_step/sec: 118.87
INFO:tensorflow:loss = 76.820786, step = 7901 (0.841 sec)
INFO:tensorflow:Saving checkpoints for 8000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:59:22
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-8000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:59:24
INFO:tensorflow:Saving dict for global step 8000: accuracy = 0.8485686, accuracy_baseline = 0.7637916, auc = 0.8982543, auc_precision_recall = 0.72675705, average_loss = 0.34301755, global_step = 8000, label/mean = 0.23620838, loss = 68.09317, precision = 0.7633588, prediction/mean = 0.21434596, recall = 0.520156
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 8000: models/census/dnn_classifier/model.ckpt-8000
INFO:tensorflow:global_step/sec: 18.8385
INFO:tensorflow:loss = 68.22385, step = 8001 (5.309 sec)
INFO:tensorflow:global_step/sec: 136.242
INFO:tensorflow:loss = 63.857376, step = 8101 (0.733 sec)
INFO:tensorflow:global_step/sec: 137.62
INFO:tensorflow:loss = 72.43463, step = 8201 (0.727 sec)
INFO:tensorflow:global_step/sec: 102.709
INFO:tensorflow:loss = 67.30856, step = 8301 (0.973 sec)
INFO:tensorflow:global_step/sec: 95.0854
INFO:tensorflow:loss = 65.98035, step = 8401 (1.052 sec)
INFO:tensorflow:Saving checkpoints for 8500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:59:31
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-8500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:59:33
INFO:tensorflow:Saving dict for global step 8500: accuracy = 0.8396609, accuracy_baseline = 0.7637916, auc = 0.8998743, auc_precision_recall = 0.73996055, average_loss = 0.33956572, global_step = 8500, label/mean = 0.23620838, loss = 67.407936, precision = 0.7665084, prediction/mean = 0.22187965, recall = 0.46189857
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 8500: models/census/dnn_classifier/model.ckpt-8500
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-8500
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575173/saved_model.pb
INFO:tensorflow:global_step/sec: 13.5099
INFO:tensorflow:loss = 61.519325, step = 8501 (7.402 sec)
INFO:tensorflow:global_step/sec: 105.772
INFO:tensorflow:loss = 67.86028, step = 8601 (0.947 sec)
INFO:tensorflow:global_step/sec: 125.784
INFO:tensorflow:loss = 62.002106, step = 8701 (0.794 sec)
INFO:tensorflow:global_step/sec: 118.336
INFO:tensorflow:loss = 62.804768, step = 8801 (0.844 sec)
INFO:tensorflow:global_step/sec: 100.356
INFO:tensorflow:loss = 68.578476, step = 8901 (0.996 sec)
INFO:tensorflow:Saving checkpoints for 9000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:59:42
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-9000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:59:44
INFO:tensorflow:Saving dict for global step 9000: accuracy = 0.8479543, accuracy_baseline = 0.7637916, auc = 0.8968927, auc_precision_recall = 0.7423756, average_loss = 0.33834887, global_step = 9000, label/mean = 0.23620838, loss = 67.16638, precision = 0.7487291, prediction/mean = 0.229197, recall = 0.5362809
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 9000: models/census/dnn_classifier/model.ckpt-9000
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-9000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575184/saved_model.pb
INFO:tensorflow:global_step/sec: 14.7885
INFO:tensorflow:loss = 60.622482, step = 9001 (6.762 sec)
INFO:tensorflow:global_step/sec: 127.27
INFO:tensorflow:loss = 59.719437, step = 9101 (0.787 sec)
INFO:tensorflow:global_step/sec: 121.027
INFO:tensorflow:loss = 65.30684, step = 9201 (0.826 sec)
INFO:tensorflow:global_step/sec: 111.639
INFO:tensorflow:loss = 77.9431, step = 9301 (0.897 sec)
INFO:tensorflow:global_step/sec: 109.415
INFO:tensorflow:loss = 67.03885, step = 9401 (0.913 sec)
INFO:tensorflow:Saving checkpoints for 9500 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-17:59:52
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-9500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-17:59:54
INFO:tensorflow:Saving dict for global step 9500: accuracy = 0.85035014, accuracy_baseline = 0.7637916, auc = 0.8994145, auc_precision_recall = 0.7478052, average_loss = 0.3350581, global_step = 9500, label/mean = 0.23620838, loss = 66.513115, precision = 0.7583425, prediction/mean = 0.22520004, recall = 0.5378414
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 9500: models/census/dnn_classifier/model.ckpt-9500
INFO:tensorflow:Performing best model export.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
INFO:tensorflow:'classification' : Classification input must be a single string Tensor; got {'hours_per_week': <tf.Tensor 'Placeholder_121:0' shape=(?,) dtype=float32>, 'workclass': <tf.Tensor 'Placeholder_127:0' shape=(?,) dtype=string>, 'relationship': <tf.Tensor 'Placeholder_126:0' shape=(?,) dtype=string>, 'gender': <tf.Tensor 'Placeholder_122:0' shape=(?,) dtype=string>, 'age': <tf.Tensor 'Placeholder_117:0' shape=(?,) dtype=float32>, 'marital_status': <tf.Tensor 'Placeholder_125:0' shape=(?,) dtype=string>, 'race': <tf.Tensor 'Placeholder_123:0' shape=(?,) dtype=string>, 'capital_gain': <tf.Tensor 'Placeholder_119:0' shape=(?,) dtype=float32>, 'native_country': <tf.Tensor 'Placeholder_129:0' shape=(?,) dtype=string>, 'capital_loss': <tf.Tensor 'Placeholder_120:0' shape=(?,) dtype=float32>, 'education': <tf.Tensor 'Placeholder_124:0' shape=(?,) dtype=string>, 'education_num': <tf.Tensor 'Placeholder_118:0' shape=(?,) dtype=float32>, 'occupation': <tf.Tensor 'Placeholder_128:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-9500
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: models/census/dnn_classifier/export/estimate/temp-1547575194/saved_model.pb
INFO:tensorflow:global_step/sec: 13.3756
INFO:tensorflow:loss = 58.544563, step = 9501 (7.476 sec)
INFO:tensorflow:global_step/sec: 85.303
INFO:tensorflow:loss = 55.173046, step = 9601 (1.172 sec)
INFO:tensorflow:global_step/sec: 90.7249
INFO:tensorflow:loss = 66.1953, step = 9701 (1.105 sec)
INFO:tensorflow:global_step/sec: 73.3921
INFO:tensorflow:loss = 70.52955, step = 9801 (1.361 sec)
INFO:tensorflow:global_step/sec: 75.481
INFO:tensorflow:loss = 77.03885, step = 9901 (1.324 sec)
INFO:tensorflow:Saving checkpoints for 10000 into models/census/dnn_classifier/model.ckpt.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-01-15-18:00:04
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from models/census/dnn_classifier/model.ckpt-10000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-15-18:00:07
INFO:tensorflow:Saving dict for global step 10000: accuracy = 0.8404595, accuracy_baseline = 0.7637916, auc = 0.9010401, auc_precision_recall = 0.75390446, average_loss = 0.33771834, global_step = 10000, label/mean = 0.23620838, loss = 67.04121, precision = 0.7943396, prediction/mean = 0.22281305, recall = 0.43797138
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10000: models/census/dnn_classifier/model.ckpt-10000
INFO:tensorflow:Loss for final step: 69.559074.
.......................................
Experiment finished at 18:00:07

Experiment elapsed time: 219.8953 seconds






    Out[157]:





<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x124afb890>



In [158]:

    
%%bash

saved_models_base=${MODEL_DIR}/export/estimate/
echo 'exported model folders:'
ls ${saved_models_base}
echo ''

saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo 'last exported model: '${saved_model_dir}
ls ${saved_model_dir}
saved_model_cli show --dir=${saved_model_dir} --all









    



exported model folders:
1547575194

last exported model: models/census/dnn_classifier/export/estimate/1547575194
saved_model.pb
variables

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['predict']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['age'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_117:0
    inputs['capital_gain'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_119:0
    inputs['capital_loss'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_120:0
    inputs['education'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_124:0
    inputs['education_num'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_118:0
    inputs['gender'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_122:0
    inputs['hours_per_week'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: Placeholder_121:0
    inputs['marital_status'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_125:0
    inputs['native_country'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_129:0
    inputs['occupation'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_128:0
    inputs['race'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_123:0
    inputs['relationship'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_126:0
    inputs['workclass'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder_127:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['class_ids'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 1)
        name: dnn/head/predictions/ExpandDims:0
    outputs['classes'] tensor_info:
        dtype: DT_STRING
        shape: (-1, 1)
        name: dnn/head/predictions/class_string_lookup_Lookup:0
    outputs['logistic'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: dnn/head/predictions/logistic:0
    outputs['logits'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: dnn/logits/BiasAdd:0
    outputs['probabilities'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 2)
        name: dnn/head/predictions/probabilities:0
  Method name is: tensorflow/serving/predict



In [ ]: